code practice
Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique
Hariharan, Suhas, Majid, Zainab Ali, Veuthey, Jaime Raldua, Haimes, Jacob
A key development in the cybersecurity evaluations space is the work carried out by Meta, through their CyberSecEval approach. While this work is undoubtedly a useful contribution to a nascent field, there are notable features that limit its utility. Key drawbacks focus on the insecure code detection part of Meta's methodology: we explore these limitations, and use our exploration as a test case for LLM-assisted benchmark analysis.